A Safety Index and Method for Flightdeck Evaluation

If our goal is to improve safety through machine, interface, and training design, then we must define a metric of flightdeck safety that is usable in the design process. Current measures associated with our notions of "good" pilot performance and ultimate safety of flightdeck performance fail to provide an adequate index of safe flightdeck performance for design evaluation purposes. The goal of this research effort is to devise a safety index and method that allows us to evaluate flightdeck performance holistically and in a naturalistic experiment. This paper uses Reason's model of accident causation (1990) as a basis for measuring safety, and proposes a relational database system and method for 1) defining a safety index of flightdeck performance, and 2) evaluating the "safety" afforded by flightdeck performance for the purpose of design iteration. Methodological considerations, limitations, and benefits are discussed as well as extensions to this work. INTRODUCTION This research is motivated by the question “How do we know if we are designing a safer flightdeck?” This effort aims to measure the degree to which a flightdeck supports good pilot performance. It is assumed that by supporting good pilot performance the goals of pilot performance, principally “safety,” are better achieved. This section reviews current approaches to measuring aviation safety and identifies requirements for a flightdeck performance safety index. Based on these considerations, I present a candidate safety index and method for evaluating flightdeck performance safety. Measuring Safety If our goal is to improve safety through machine, interface, and training design, then we must define a metric of flightdeck safety that is usable in the design process. Accident rate, in terms of hull losses or fatalities, has been used as a measure of aviation system safety. However, accidents are extremely rare occurrences, and therefore are not a meaningful metric for evaluating the safety of a particular flight/flightdeck/operator mission. Incidents, including regulatory violations, while more frequent than accidents are still relatively rare (Wickens, 1995, p. 126). Typical human performance experiments in aviation assess the degree to which a particular system, procedure, or training regime improves safety by measuring a narrow band of performance measures directly related to that intervention. These focused experiments typically include system-specific errors and a few particular reaction times as dependent measures. There are several problems with using this approach. First, subjects manage trade-offs between reaction-time performance and accuracy (Pachella, 1974). Second, these finer grained measures can be of questionable operational significance. For example, faster reaction times do not always translate to behavior consistent with improved safety (cf. Rogers, Schutte and Latorella, 1996). Finally, in these focused experiments, you frequently get what you measure; that is, subjects manage resources to optimize performance on those aspects that they believe are being measured. Subjective assessments are also commonly used in focused experiments. However, subjective evaluations may dissociate from actual performance (Yeh & Wickens, 1988) and may be unfairly biased towards familiar designs. Physiological measures of stress and arousal only directly indicate intermediary states, and are therefore also removed from an operational definition of safe performance. In summary, current measures associated with our notions of "good" pilot performance and ultimate safety of flightdeck performance fail to provide an adequate index of safe flightdeck performance for design evaluation purposes. Requirements for a Safety Index The goal of this research effort is to devise a safety index and method that allows us to evaluate flightdeck performance holistically and in a naturalistic experiment (Beach et al., 1997). This safety index must provide sufficient data to indicate levels of safety achieved in flightdeck performance. The index must be traceable to conventional notions of safety in aviation to support face and construct validity. The safety index should be sensitive; that is, it should provide an adequate range of magnitude to reflect differences in what is measured. It should be unobtrusive and nonreactive; that is, it should not interfere with the natural performance of the task, and it should not be obvious to the subject what is being measured. By ensuring that the measure is not obtrusive or invasive and does not induce biases to "game" performance, one encourages subjects' motivational and goal structures in the real environment to be retained in the experimental environment. Finally, the index must be generalizable; that is, usable across classes of flightdecks for comparison purposes, and tailored to a particular mission. The safety index, to be a worthy measure, must also posess predictive validity and be reliable.